This invention relates generally to user interfaces for computer systems and, more particularly, to a user interface system and method for automated optical inspection systems.
The current generation of computer vision systems and, more specifically, computer vision systems used for automated optical inspection (xe2x80x9cAOIxe2x80x9d), place a significant burden on the operators of these systems to develop programs that will enable the system to classify an image into one or more disjoint or non-disjoint classes. Because of this, the user interface elements associated with these systems present to their users options for programming or modifying the system""s existing program. This makes current AOI systems exceedingly difficult to use. To increase ease of use, such systems incorporate a graphical user interface (xe2x80x9cGUIxe2x80x9dxe2x80x94pronounced xe2x80x9cgooeyxe2x80x9d). GUIs have become more prevalent in such systems with the increasingly widespread availability of powerful microprocessors.
A GUI is a type of display format that enables a user to operate a computer controlled system by pointing to pictorial representations, such as xe2x80x9cwindowsxe2x80x9d and xe2x80x9ciconsxe2x80x9d (bitmaps), on a display device. A window is a rectangle displayed on a screen that affords a user workspace within a program. In a typical operation, the user may move the window about on the screen, change its size or shape, enlarge it to fill the screen, close it entirely, or change how much of its contents are displayed. To aid the user in the manipulation of its contents, a window will typically include a number of user interface components, such as buttons, menus, sliders, and the like. Outside the window, the screen can display other screen objects, such as other windows, or related computer system representations, such as disk drive icons or toolbars.
To navigate within a GUI, most systems employ a screen cursor or pointer, typically displayed as a small arrow icon (bitmap) which allows the user to select individual points on the screen. In operation, the screen cursor is moved to a desired screen location in response to the movement of a pointing device (e.g., a mouse) by the user. Besides effecting cursor movement, most pointing devices include one or more switches or xe2x80x9cmouse buttonsxe2x80x9d for specifying additional user input or xe2x80x9cuser eventsxe2x80x9d by xe2x80x9cclickingxe2x80x9d on (selecting) objects in the display device. Since many user choices may be entered through use of a pointing device (e.g., for selecting screen objects), instead of input with the keyboard, the need for the user to memorize special commands is lessened.
GUIs feature a menu bar, for instance, running across the top of the screen which serves to group or categorize commands available to the user. Clicking on an item on the menu bar typically causes a xe2x80x9cpulldown menuxe2x80x9d to appear. This second or xe2x80x9csub-menuxe2x80x9d also includes a number of items, each of which is associated with a desired action, including the display of even more menus. To select a desired action, the user usually clicks the corresponding menu item with the screen or mouse pointer. For some menu items, particularly those which may be nested in several layers deep, a keyboard equivalent or xe2x80x9chotkeyxe2x80x9d may be available. The conventional graphical user interface described above significantly reduces the amount of information that a user must recall in order to effectively use the computer controlled system.
Current AOI systems incorporate graphical user interfaces for system users to interact with and modify the AOI program. However, current graphical user interfaces for AOI systems present information in a static manner to the user and the user cannot (or finds it difficult to) interact with or explore the data presented in greater depth. The information displayed to the user by the GUI in current AOI systems is generally not linked to other pieces of information on the same interface in a way that the user can manipulate the information to get still more information. For example, the user cannot filter the data or provide information to the system via the GUI.
One existing method for modifying an AOI program is through the use of automated threshold modification functions. These functions modify the behavior of an AOI system program by analyzing the AOI algorithm performance with different threshold values with respect to a set of test images. However, this method is merely an extension of the basic AOI programming methodology and, as such, still has the same efficiency and effectiveness problems normally associated with those systems. The operator is using a programming tool rather than a user interface tool and is imparting programming information rather than information that can be used to distinguish between possible class memberships of the image under inspection.
Current user interfaces, therefore, do not allow information to be imparted to the AOI system through the interacting behavior with the information displayed in the user interface. The control elements in such a graphical user interface cannot be manipulated and simultaneously updated because they are not linked in the underlying programming. Current AOI system GUIs only display information to the operator/user, but the user cannot make queries or organize the information for viewing in different ways. Additionally, by virtue of not being able to interact with the system, the user cannot increase the performance of the AOI system by imparting information learned about the production process back to the AOI system.
As a result, currently existing AOI system GUIs do not provide the user the capability to fine tune and troubleshoot false calls and defect occurrences for diagnosing a problem on the line. For example, in order to reduce the scope of a specific inspection algorithm of the AOI system, current systems require the modification of the algorithm for the entire component class(es) the algorithm looks at, instead of allowing a component-by-component basis modification. The user is therefore unable to go into the system and very quickly determine what process areas may be suspicious. Heavy reliance on process engineers is thus required in prior art AOI systems lacking a more interactive user interface.
Similarly, prior art AOI system GUIs do not provide the capability to filter defect classifications by different parameters (for example decision confidence values) that are linked in such a way as to allow narrowing of the data presented to point to a specific line problem or defect. A user of such a prior art system cannot easily determine and corroborate the reliability of classification decisions made by the system. Additionally, using current AOI system GUIs, a user cannot compare a current board defect to, for example, a prior closest example of that defect. Such a comparison allows a user to quickly and with more confidence conceptualize the decision as to whether a defect has been correctly classified. Such a comparison further provides for increased confidence in the continuity of interpretation of defects between different users. By comparing the image of a current defective board to images of prior boards and defects and, more particularly, to the closest identified defect to the current board, the user can perform a selection task instead of a recognition task. The user is thus not forced to look at an image and determine the existence of a defect without a basis for comparison. Instead, the user decides whether the current board defect is similar to a prior detected defect of high-confidence.
Presently existing AOI system GUIs also do not give a user the capability to impart user corroboration of process performance back to a system capable of learning. In this regard, we make reference to pending related U.S. patent application Ser. No. 09/935,531, filed on Sep. 22, 1997, which discloses an AOI algorithm capable of learning through both automated and manual input as to the confidence level to impart to a detected defect. If an AOI system had the underlying capability to learn from user input, current graphical user interfaces do not allow for real-time training of the AOI system because they do not provide the capability for a user to easily impart knowledge to the system.
In particular, prior art systems do not allow the user to impart information to the system in such a way that the system learns and assigns reliable confidence values to defects as they occur, i.e., the user cannot update the system knowledge-base on a real-time basis. As a result, current AOI systems defect recognition cannot be improved in real time to more reliably recognize subsequent defects or to treat subsequent defects in a particular manner. Relatedly, particular types of defects cannot be tracked to determine when they change, or to ignore them in certain cases. This capability is useful, for example, when using equivalent alternate parts from different manufacturers.
Initial training of an AOI system via a graphical user interface is also currently a non-trivial process. Graphical user interfaces on current AOI systems do not have an easy-to-learn and easy-to-use process for a user to initially train the AOI system in defect recognition. Thus, the underlying learning of the present invention provides the capability to use an extremely simple (xe2x80x9cwizardxe2x80x9d) interface to train the system. In contrast, prior art systems cannot because the underlying systems are incapable of learning (i.e., require programming). Additionally, prior art systems do not provide the ability to control access to sensitive menu settings.
Therefore a need has arisen for a user interface for an automated optical inspection system that is both easy to use and provides the operator with the capability to impart information back into the automated optical inspection system.
A further need exists for an automated optical inspection system user interface that dynamically links the information items displayed within different areas of the user interface such that a user can interact within one view area and simultaneously link to and/or update the corresponding information items in another view area.
A still further need exists for an AOI system user interface having the capability to discriminate (xe2x80x9cfilterxe2x80x9d) the type and amount of information displayed in the view areas of the user interface.
An even further need exists for an AOI system user interface that allows the user to train the AOI system in real-time, and without interruption of the inspection process, to increase the confidence level in the defect detection capability of the AOI system and to improve the AOI system""s identification and classification of defects.
Yet a further need exists for an AOI system user interface that allows the user to troubleshoot the AOI system process to determine the source of false calls and defect occurrences in the process without having to modify the entire inspection algorithm for the entire component class.
Further still, a need exists for an AOI system user interface with the capability to present to the user a side-by-side visual comparison of a currently detected board defect and a closest high-confidence defect match, or a visual example of what the component should look like, from the AOI systems knowledge-base to provide for a rapid and more reliable determination of the existence of a defect by a user.
A still further need exists for an AOI system user interface having the capability for a user to quickly and accurately determine the source of process errors, thereby allowing economically feasible small production runs of different boards and/or components without the heavy reliance on highly experienced process engineers (as in prior art such systems).
Even further, a need exists for an AOI system user interface that provides access controls to protect sensitive menu items.
In accordance with the present invention, an AOI user interface is provided that substantially reduces or eliminates the disadvantages and problems associated with previously developed AOI user interfaces, including the problems of extensive reliance on process engineers, the inability to filter linked information parameters for process troubleshooting and defect identification, the inability to easily impart information back to the AOI system in real-time to update the AOI system""s knowledge-base, and the lack of a dynamic link between information items in the user interface from one view area to another.
More specifically, the present invention provides a user interface method and system allowing for user selection on a display device of one or more functions performed via a computer in an automated optical inspection system. The user interface of the present invention provides a real-time information display for a supervising line operator so that he or she can readily observe critical board inspection information and potential undesirable operating conditions and rapidly initiate corrective action. The user interface includes one or more button icons respectively associated with the user selectable functions. The user interface also includes at least one view area window for displaying information items, wherein the information items displayed in the window are dynamically linked to one another to provide interactivity between windows and simultaneous updating of related information items in all views. A filter area window is provided for the user to select which information items to discriminate out and display in the view windows. The filter area window is operable to simultaneously update the information items displayed in each of the other windows in the user interface based on the discrimination criteria selected by the user in the filter area.
The AOI system user interface method and system of the present invention can be a graphical user interface having view windows in which either text, graphics or both are displayed simultaneously. The information items displayed in each of the windows can be interrelated such that, for example, if the information item in one window is a textual description of a board defect or properties of board defects, a corresponding information item in another view area window can be a graphical representation of that same defect. If a user selects a textual description of a defect, a graphical representation of the selected defect can be made to appear in a related window. Likewise, when a user selects a discrimination criteria in the filter area window, the data meeting the discrimination criteria can appear in either textual, graphical, or both formats in corresponding view windows. The automated optical inspection user interface method and system of the present invention can have multiple view area windows with multiple functions associated between them.
As a technical advantage, the automated optical inspection user interface and method of this invention provides a user interface for an automated optical inspection system that is both easy to use and provides the operator with the capability to impart defect classification information back into the automated optical inspection system.
A further technical advantage of the present invention is that it provides an automated optical inspection system user interface that dynamically links the information items displayed within different areas of the user interface such that a user can interact with information items in one view area and simultaneously link to and/or update the corresponding information items in another view area.
A still further technical advantage is that the AOI system user interface of this invention has the capability to discriminate (xe2x80x9cfilterxe2x80x9d) the type and amount of information displayed in the view areas of the user interface.
An even further technical advantage is that the AOI system user interface of this invention allows the user to train the AOI system in real-time, and without interruption of the inspection process, to increase the confidence level in the defect detection capability of the AOI system and to improve the AOI system""s identification and classification of defects.
A further technical advantage of the AOI system user interface of the present invention is that the user can troubleshoot the AOI system process to determine the source of false calls and defect occurrences in the process without having to modify the entire inspection algorithm to reduce the scope of said inspection algorithm to a single component.
Yet another technical advantage of the AOI system user interface is the ability to present to the user a side-by-side visual comparison of a currently detected or identified board defect and a closest high-confidence defect match, or a visual example of what the component should look like, from the AOI systems knowledge-base to provide for a rapid and more reliable determination of the existence of a defect by a user.
Still a further technical advantage is the ability of the AOI system user interface of this invention to permit a user to quickly and accurately determine the source of process errors, thereby allowing economically feasible small production runs of different boards and/or components without the heavy reliance on process engineers (as in prior art systems).
An even further technical advantage is the ability of the AOI system user interface of this invention to provide access controls to protect sensitive menu items.